Color scannerless range imaging system using an electromechanical grating

ABSTRACT

A scannerless range imaging system includes an illumination system and an electromechanical light modulator. The illumination system illuminates objects in the scene with modulated illumination of a predetermined modulation frequency, and the modulated illumination reflected from objects in the scene incorporates a phase delay corresponding to the distance of the objects from the range imaging system. The electromechanical light modulator, which is positioned in an optical path of the reflected illumination, operates at a reference frequency that corresponds to the predetermined modulation frequency and accordingly modulates the modulated illumination reflected from the object, thereby generating a phase image from the interference between the reference frequency and the reflected modulated illumination. The system further includes an optical system that deflects the reflected illumination to the electromechanical light modulator and redirects the phase image generated by the electromechanical light modulator to an image capture section. A color image may be captured by moving the optical system out of the optical path such that reflected illumination from the object will pass directly to the image capture section without contacting the electromechanical light modulator.

FIELD OF THE INVENTION

The present invention relates to the field of three-dimensional imagecapture and in particular to techniques for modulating the reflectedlight in order to extract phase images and for capturing a color textureimage in conjunction with the phase image.

BACKGROUND OF THE INVENTION

Distance (or depth) information from a camera to objects in a scene canbe obtained by using a scannerless range imaging system having amodulated illumination source and a modulated image receiver. In amethod and apparatus described in U.S. Pat. No. 4,935,616 (and furtherdescribed in the Sandia Lab News, vol. 46, No. 19, Sep. 16, 1994), ascannerless range imaging system uses either an amplitude-modulatedhigh-power laser diode or an array of amplitude-modulated light emittingdiodes (LEDs) to simultaneously illuminate a target area. Conventionaloptics confine the target beam and image the target onto a receiver,which includes an integrating detector array sensor having hundreds ofelements in each dimension. The range to a target is determined bymeasuring the phase shift of the reflected light from the targetrelative to the amplitude-modulated carrier phase of the transmittedlight. To make this measurement, the gain of an image intensifier (inparticular, a micro-channel plate) within the receiver is modulated atthe same frequency as the transmitter, so the amount of light reachingthe sensor (a charge-coupled device) is a function of therange-dependent phase difference. A second image is then taken withoutreceiver or transmitter modulation and is used to eliminatenon-range-carrying intensity information. Both captured images areregistered spatially, and a digital processor is used to extract rangedata from these two frames. Consequently, the range associated with eachpixel is essentially measured simultaneously across the whole scene.

The scannerless range imaging system described above utilizes an imageintensifier (specifically, a micro-channel plate) of the type producedby Litton Industries. The primary purpose of the intensifier is toprovide a reference frequency to operate upon the modulated light signalfrom the illuminator that is reflected from the target. By modulatingthe gain of the image intensifier the reflected, modulated light signalis multiplied by the intensifier gain and constructive and destructiveinterference is established. A primary application of the scannerlessrange imaging system is to enable a method of creating a virtualthree-dimensional environment from photographs. While range data is animportant part of this application, a so-called texture image is alsoneeded. The texture image should ideally be captured with identicaloptical properties as the range data to assure proper registrationbetween range and texture values. Furthermore, having a color textureimage is highly desirable for many practical and commercialapplications.

A drawback of methods using an image intensifier is that colorinformation is lost. Unfortunately for color applications, an imageintensifier operates by converting photonic energy into a stream ofelectrons, amplifying the energy of the electrons and then convertingthe electrons back into photonic energy via a phosphor plate. Oneconsequence of this process is that color information is lost. Sincecolor is a useful property of images for many applications, a means ofacquiring the color information that is registered along with the rangeinformation is extremely desirable.

One approach to acquiring color is to place a dichromatic mirror in theoptical path before the micro-channel-plate. Following the mirror aseparate image capture plane (i.e., a separate image sensor) is providedfor the range portion of the camera and another image capture plane(another sensor) is provided for the color texture capture portion ofthe camera. This is the approach taken by 3DV Technology with theirZ-Cam product. Besides the added expense of two image capture devices,there are additional drawbacks in the need to register the two imageplanes precisely, together with alignment of the optical paths. Anotherdifficulty is collating image pairs gathered by different sources.

Recognizing that the system described in the '616 patent may beimplemented in relation to a normal camera system, and, in particular,that a standard camera system may be converted into a range capturesystem by modifying its optical system, another approach is to employinterchangeable optical assemblies: one optical assembly for the phaseimage portion and a separate optical element for the color texture imageportion. This approach is described in detail in commonly assignedcopending application Ser. No. 09/451,823, entitled “Method andApparatus for a Color Scannerless Range Image System” and filed Nov. 30,1999 in the names of Lawrence A. Ray, Louis R. Gabello and Kenneth J.Repich. The drawback of this approach is the need to switch lenses andthe possible misregistration that might occur due to the physicalexchange of lens elements. There is an additional drawback in the timerequired to swap the two optical assemblies, and the effect that mayhave on the spatial coincidence of the images.

In commonly-assigned, copending U.S. patent application Ser. No.09/572,522, now U.S. Pat. No. 6,349,174 B1 entitled “Method andApparatus for a Color Scannerless Range Imaging System” and filed May17, 2000 in the names of Lawrence A. Ray and Louis R. Gabello, abeamsplitter located in the primary optical path separates the reflectedimage light into two channels, a first channel including an infraredcomponent and a second channel including a color texture component,whereby one of the channels traverses a secondary optical path distinctfrom the primary path. A modulating element, i.e., an intensifier, isoperative in the first channel to receive the infrared component and amodulating signal, and to generate a processed infrared component withphase data indicative of range information. An optical network isprovided in the secondary optical path for recombining the secondaryoptical path into the primary optical path such that the processedinfrared component and the color texture component are directed to theimage responsive element. This technique eliminates the requirement fortwo image capture planes, as well as for interchangeable opticalassemblies, and allows the operator to collect a full range map withtexture with a single exposure activation.

In addition to the loss of color information, and the consequentnecessity to devise techniques as described above to overcome thisdrawback, the image intensifier is a costly part and, in addition, canbe fragile. In order to reduce the cost of the scannerless range imagingsystem, a less expensive alternative technology would be attractive.Since a primary purpose of the image intensifier is to act as amodulating shutter, an alternative technology will have to perform thistask. What is needed is an alternative technology that would avoid theaforementioned limitations; in addition, it would be desirable tocapture ranging information without sacrificing color information thatwould otherwise be available for capture.

SUMMARY OF THE INVENTION

It is an object of the invention to provide an alternative technology toan image intensifier for the receiver modulation function in ascannerless range imaging system.

It is a further object of the invention to capture a color texture imageas well as one or more phase images on the same image plane for eachpoint on the image.

The present invention is directed to achieving these objectives whileovercoming one or more of the problems set forth above. Brieflysummarized, according to one aspect of the present invention, ascannerless range imaging system for capturing range information of ascene includes an illumination system and an electromechanical lightmodulator. The illumination system illuminates objects in the scene withmodulated illumination of a redetermined modulation frequency, wherebythe modulated illumination reflected from objects in the sceneincorporates a phase delay corresponding to the distance of the objectsfrom the range imaging system. The electromechanical light modulator,which is positioned in an optical path of the modulated illuminationreflected from the object, operates at a reference frequency thatcorresponds to the predetermined modulation frequency and accordinglymodulates the modulated illumination reflected from the object, therebygenerating an image from the interference between the referencefrequency and the reflected modulated illumination. This captured image,which is thereafter referred to as a phase image, is used to deriverange data. An image capture section, also positioned in the opticalpath of the modulated illumination reflected from the object, capturesthe phase image.

Since the electromechanical light modulator operates via modulatingelements having reflective surfaces, the system further includes anoptical system having a mirror element that deflects the reflectedmodulated illumination upon the reflective surfaces of theelectromechanical light modulator and redirects the phase imagereflected from the reflective surfaces of the electromechanical lightmodulator to the image capture section. A color image may be captured bymoving the optical system out of the optical path such that reflectedillumination from the object will pass directly to the image capturesection without contacting the electromechanical light modulator. Apreferred electromechanical light modulator is an electromechanicalgrating.

Consequently, an advantage of the invention is that it provides both analternative to the intensifier and a simplified technique for capturinga color texture image as well as one or more phase images. A furtheradvantage of the invention is that it eliminates the need for anexpensive component, i.e., the intensifier, with a device that issignificantly less expensive and less fragile. The electromechanicalgrating is also lighter in weight and more compact than a micro-channelplate, and uses a lower operating voltage. Moreover, the system is ableto capture a color image in addition to the phase images without thesort of clever work-arounds shown in the prior art. The system is alsoable to operate in a continuous modulation mode or in a pulse mode.

These and other aspects, objects, features and advantages of the presentinvention will be more clearly understood and appreciated from a reviewof the following detailed description of the preferred embodiments andappended claims, and by reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a system level diagram of a scannerless range imaging systemin accordance with the invention.

FIG. 2 is a diagram illustrating an image bundle and related datacaptured by the system shown in FIG. 1.

FIG. 3 is a diagram showing more detail of the illumination system shownin FIG. 1.

FIG. 4 is a diagram showing more detail of the electromechanical gratingshown in FIG. 1.

FIG. 5 is a diagram showing more detail of the image capture systemshown in FIG. 1.

FIG. 6 is a block diagram of a known range imaging system which can beused to capture a bundle of images.

DETAILED DESCRIPTION OF THE INVENTION

Because range imaging devices employing laser illuminators and capturedevices including image intensifiers and electronic sensors are wellknown, the present description will be directed in particular toelements forming part of, or cooperating more directly with, apparatusin accordance with the present invention. Elements not specificallyshown or described herein may be selected from those known in the art.Certain aspects of the embodiments to be described may be provided insoftware. Given the system as shown and described according to theinvention in the following materials, software not specifically shown,described or suggested herein that is useful for implementation of theinvention is conventional and within the ordinary skill in such arts.

It is helpful to first review the principles and techniques involved inscannerless range imaging, as known in the prior art. Accordingly,referring first to FIG. 6, a known scannerless range imaging system 10is shown as a laser radar that is used to illuminate a scene 12 and thento capture an image bundle comprising a minimum of three images of thescene 12. An illuminator 14 emits a beam of electromagnetic radiationwhose frequency is controlled by a modulator 16. Typically, in the priorart, the illuminator 14 is a laser device which includes an opticaldiffuser in order to effect a wide-field illumination. The modulator 16provides an amplitude varying sinusoidal modulation. The modulatedillumination source is modeled by:

L(t)=μ_(L)+η sin(2πƒt)  (Eq. 1)

where μ_(L) Is the mean illumination, η is the modulus of theillumination source, and ƒ is the modulation frequency applied to theilluminator 14. The modulation frequency is sufficiently high (e.g.,12.5 MHz) to attain sufficiently accurate range estimates. The outputbeam 18 is directed toward the scene 12 and a reflected beam 20 isdirected back toward a receiving section 22. As is well known, thereflected beam 20 is a delayed version of the transmitted output beam18, with the amount of phase delay being a function of the distance ofthe scene 12 from the range imaging system. Typically, in the prior art,the reflected beam 20 strikes a photocathode 24 within an imageintensifier 26, thereby producing a modulated electron streamproportional to the input amplitude variations. The gain modulation ofthe image intensifier 26 is modeled by:

M(t)=μ_(M)+γ sin(2πƒt)  (Eq. 2)

where μ_(M) is the mean intensification, γ is the modulus of theintensification and ƒ is the modulation frequency applied to theintensifier 26. The purpose of the image intensifier is not only tointensify the image, but also to act as a frequency mixer and shutter.Accordingly, the image intensifier 26 is connected to the modulator 16,causing the gain of a microchannel plate 30 to modulate. The electronstream from the photocathode 24 strikes the microchannel plate 30 and ismixed with a modulating signal from the modulator 16. The modulatedelectron stream is amplified through secondary emission by themicrochannel plate 30. The intensified electron stream bombards aphosphor screen 32, which converts the energy into a visible lightimage. The intensified light image signal is captured by a capturemechanism 34, such as a charge-coupled device (CCD). The captured imagesignal is applied to a range processor 36 to determine the phase delayat each point in the scene. The phase delay term ω of an object at arange ρ meters is given by: $\begin{matrix}{\omega = \frac{4\quad \pi \quad \rho \quad f}{c}} & \left( {{Eq}.\quad 3} \right)\end{matrix}$

where c is the velocity of light in a vacuum. Consequently, thereflected light at this point is modeled by:

R(t)=μ_(L)+κ sin(2πƒt+ω)  (Eq. 4)

where κ is the modulus of illumination reflected from the object. Thepixel response P at this point is an integration of the reflected lightand the effect of the intensification: $\begin{matrix}{P = {{\int_{0}^{2\quad \pi}{{R(t)}{M(t)}\quad {t}}} = {{2\quad \mu_{L}\mu_{M}} + {\kappa \quad {\pi\gamma}\quad \cos \quad (\omega)}}}} & \left( {{Eq}.\quad 5} \right)\end{matrix}$

In the range imaging system disclosed in the aforementioned '616 patent,a reference image is captured during which time the micro-channel plateis not modulated, but rather kept at a mean response. The range isestimated for each pixel by recovering the phase term as a function ofthe value of the pixel in the reference image and the phase image.

A preferred, more robust approach for recovering the phase term isdescribed in commonly-assigned U.S. Pat. No. 6,118,946, entitled “Methodand Apparatus for Scannerless Range Image Capture Using PhotographicFilm”, which is incorporated herein by reference. Instead of collectinga phase image and a reference image, this approach collects at leastthree phase images (referred to as an image bundle). This approachshifts the phase of the intensifier 26 relative to the phase of theilluminator 14, and each of the phase images has a distinct phaseoffset. For this purpose, the range processor 36 is suitably connectedto control the phase offset of the modulator 16, as well as the averageillumination level and such other capture functions as may be necessary.If the image intensifier 26 (or laser illuminator 14) is phase shiftedby θ_(i), the pixel response from equation (5) becomes:

P _(i)=2μ_(L)μ_(M)π+κπγ cos(ω+θ_(i))  (Eq. 6)

It is desired to extract the phase term ω from the expression. However,this term is not directly accessible from a single image. In equation(6) there are three unknown values and the form of the equation is quitesimple. As a result, mathematically only three samples (from threeimages) are required to retrieve an estimate of the phase term, which isproportional to the distance of an object in the scene from the imagingsystem. Therefore, a set of three images captured with unique phaseshifts is sufficient to determine ω. For simplicity, the phase shiftsare given by θ_(k) =2πκ/3, κ=0,1,2. In the following description, animage bundle shall be understood to constitute a collection of imageswhich are of the same scene, but with each image having a distinct phaseoffset obtained from the modulation applied to the intensifier 26. Itshould also be understood that the analysis can also be performed byphase shifting the illuminator 14 instead of the intensifier 26. If animage bundle comprising more than three images is captured, then theestimates of range can be enhanced by a least squares analysis using asingular value decomposition (see, e.g., W. H. Press, B. P. Flannery, S.A. Teukolsky and W. T. Vetterling, Numerical Recipes (the Art ofScientific Computing), Cambridge University Press, Cambridge, 1986).

If images are captured with n≧3 distinct phase offsets of theintensifier (or laser or a combination of both) these images form animage bundle. Applying Equation (6) to each image in the image bundleand expanding the cosine term (i.e., P_(i)=2,μ_(L)μ_(M)π+κπγ(cos(ω)cos(θ_(i))−sin(ω)sin(θ_(i)))) results in thefollowing system of linear equations in n unknowns at each point:$\begin{matrix}{\begin{pmatrix}P_{1} \\P_{2} \\\vdots \\P_{n}\end{pmatrix} = {\begin{pmatrix}1 & {\cos \quad \theta_{1}} & {{- \sin}\quad \theta_{1}} \\1 & {\cos \quad \theta_{2}} & {{- \sin}\quad \theta_{2}} \\\vdots & \vdots & \vdots \\1 & {\cos \quad \theta_{n}} & {{- \sin}\quad \theta_{n}}\end{pmatrix}\quad \begin{pmatrix}\Lambda_{1} \\\Lambda_{2} \\\Lambda_{3}\end{pmatrix}}} & \left( {{Eq}.\quad 7} \right)\end{matrix}$

where Λ=2μ_(L)μ_(M)π, Λ₂=κπγ cos ω, and Λ₃=κπγ sin ω. This system ofequations is solved by a singular value decomposition to yield thevector Λ=[Λ₁, Λ₂, Λ₃]^(τ). Since this calculation is carried out atevery (x,y) location in the image bundle, Λ is really a vector imagecontaining a three element vector at every point. The phase term ω iscomputed at each point using a four-quadrant arctangent calculation:

ω=tan⁻¹(Λ₃, Λ₂)  (Eq. 8)

The resulting collection of phase values at each point forms the phaseimage. Once phase has been determined, range p can be calculated by:$\begin{matrix}{\rho = {\omega \frac{c}{4\quad \pi \quad f}}} & \left( {{Eq}.\quad 9} \right)\end{matrix}$

Equations (1)-(9) thus describe a method of estimating range using animage bundle with at least three images (i.e., n=3) corresponding todistinct phase offsets of the intensifier and/or illuminator.

What the present invention specifically addresses is an alternativetechnology to the image intensifier 26, as used in the prior art. Thepreferred alternative technology is an electromechanical grating, whichis in the class of electromechanical light modulators. Theelectromechanical grating is a device with a periodic sequence ofreflective elements that form electromechanical phase gratings. In suchdevices, the incident beam is selectively reflected or diffracted into anumber of discrete orders. Depending on the application, one or more ofthese diffracted orders may be collected and used by the optical system.

An electromechanical grating with a fast response time is a binaryelectromechanical grating made of suspended micro-mechanical ribbonelements as described in Bloom et al., “Method and Apparatus forModulating a Light Beam,” U.S. Pat. No. 5,311,360, issued May 10, 1994.This device, also known as a grating light valve (GLV), can befabricated with CMOS-like processes on silicon. Improvements in thedevice were later described by Bloom, et al. that included: 1) patternedraised areas beneath the ribbons to minimize contact area to obviatestiction between the ribbons and the substrate, and 2) an alternativedevice design in which the spacing between ribbons was decreased andalternate ribbons were actuated to produce good contrast (see Bloom, etal., “Deformable Grating Apparatus for Modulating a Light Beam andIncluding Means for Obviating Stiction Between Grating Elements andUnderlying Substrate,” U.S. Pat. No. 5,459,610, issued Oct. 17, 1995).An alternative electromechanical grating with a partially conformalgrating structure and a potentially higher fill factor was described byKowarz in “Spatial Light Modulator with Conformal Grating Elements,”U.S. patent application Ser. No. 09/491,354, filed Jan. 26, 2000 (CIPSer. No. 09/867,927 filed May 30, 2001). The disclosures of each ofthese patents, and the patent applications, are incorporated herein byreference.

As will be clear to those of ordinary skill in this art, when theelectromechanical grating is used in place of the intensifier, themethod of estimating range using an image bundle will remain essentiallythe same as described above in relation to FIG. 6 except that the outputM(t) of the modulating element represented by Eq. (2) would be modifiedto represent the effect produced by the electromechanical grating. Thismodification would carry through the remaining equations, although thebasic logic and the model would remain the same. In particular, theprinciples for range determination remain exactly the same, that is, therange is determined using an image bundle with at least three images(i.e., n=3) corresponding to distinct phase offsets of the modulatorand/or illuminator.

Referring now to FIG. 1, the overall scannerless range imaging (SRI)system is shown as a range camera comprised of a number of subsystems,including a controller 40, an illuminator 42, a lens/shutter combination44, an electromechanical grating light modulator 46, a mirror 47 and animage capture subsystem 48. The image capture subsystem 48 includes aphotosensor, e.g., a photosensitive film or an electronic sensor, suchas a charge-coupled device (CCD). The controller 40 manages the workflowwithin the device, sequences the events and establishes one or morebaseline system frequencies. The illuminator 42 emitsamplitude-modulated light, preferably in the infrared band. Thecontroller 40 also has the ability to phase shift the modulation signalto the illuminator 42 relative to a reference modulation within thecontroller 40. The lens/shutter 44 controls the image focal length, thefield of view and other normal properties of photographic lenses andshutters. The electromechanical grating 46 operates at a referencefrequency that is managed by the controller 40. The purpose of theelectromechanical grating 46 is to mix a reference frequency with thereflected light from objects in the scene. The electromechanical grating46 is tuned to have maximum efficiency at the same wavelength as emittedby the illuminator. The image capture subsystem 48 records the framescaptured by the SRI (Scannerless Range Imaging) system for subsequentprocessing. In the preferred embodiment, a single image capture planeresponsive to the infrared spectrum is employed for the photosensitiveelement; in addition, if a color texture image is to be captured, thephotosensitive element must be capable of responding to light in thevisible spectrum.

The illumination and reception aspect of the SRI system shown in FIG. 1generally operates as described in connection with the known systemshown in FIG. 6, that is, an output beam 43 a is directed toward a sceneand a reflected beam 43 b is directed back toward the receiving section,which includes the lens/shutter combination 44, the electromechanicalgrating 46 and the image capture subsystem 48. As is well known, thereflected beam 43 b is a delayed version of the transmitted output beam43 a, with the amount of phase delay being a function of the distance ofthe scene from the SRI system. Unlike the prior art, the reflected beam43 b is deflected from the system optical path 43 c and upon theelectromechanical grating 46, thereby producing a modulated light imagesignal. The modulated light image signal is then deflected back into theoptical path 43 c and captured by the image capture subsystem 48, suchas a charge-coupled device (CCD). Though not shown specifically in FIG.1, the captured image signal is applied to a range processor of the typeshown in FIG. 6 to determine the phase delay at each point in the scene.The foregoing system is sufficient to capture a phase image. If a colortexture image is also to be captured, the mirror 47 is retracted out ofthe optical path and the color texture image is directly transmitted tothe image capture subsystem 48.

The controller 40 is the overall device manager and has the task ofcommunicating with the subsystems to sequence events, to provide properpower to the devices and to provide a common synchronizing frequency.Subsequent descriptions of each subsystem include the interface of thesubsystem with the controller. The controller is the primary interfaceof a user with the system. A user need only trigger the device once andthe system creates a complete image bundle.

As shown in relation to FIG. 2, the notion of an image bundle 50 iscentral to the range estimation method. The image bundle 50 includes acombination of images 52, 54 captured by the system as well asinformation (bundle data 56) pertinent to the individual images andinformation common to all the images. The image bundle contains twotypes of images: a set of phase images 52 related to the range captureportion of the process and a color image 54, commonly referred to as thetexture image. For the set of phase images, each phase image is acquiredwhile the illuminator 42 is operating with a phase offset from thereference frequency supplied by the controller 40. The color image 54 isacquired when the electromechanical grating 46 is inactive. Commoninformation in the image bundle data 56 would typically include thenumber of phase images in the bundle (three or more) and the modulationfrequency utilized by the camera system. Other information might be thefocal length of the lens, the size of the image plane, the number ofhorizontal and vertical pixels in the images, the field-of-view of theimaging system and/or data related to camera status at the time of theimage capture. This information will be used in subsequent processing toconvert the image values into three-dimensional positions of each pixelin space relative to the camera location and orientation.

Image specific information in the image bundle data 56 will include thephase offset 1. . . N used for each (1. . . N) of the individual phaseimages 52. The image bundle 50 includes a minimum of three such images,each of which are monochrome. Each of the phase images 52 records theeffect of a distinct phase offset applied to either the illuminationsystem 42 or the electromechanical grating 46. The additional colorimage 54 is an image that does not contain range capture components,instead containing the color texture information in the actual image.Although this is a color image, it is preferably, but not necessarily,the same size as the phase images 52.

The illuminator 42 shown in FIG. 3 has the primary purpose of producingan amplitude-modulated illumination with its phase controllable forgenerating a shift in the transmitted wave pattern for each phase image52 in the image bundle 50 (although, as mentioned before, this functionmay be performed by modulation of the reflected illumination in thecapture portion of the color scannerless range imaging system). Theilluminator 42 includes a light source, which is preferably a laserlight source 60 operating in the infrared band with a power outputintensity of about 0.5 watt, and a modulation circuit 62 controllablethrough a line 64 from the controller 40 (see FIG. 1), for generatingthe requisite modulation signals of predetermined frequency with a setof predetermined phase offsets. The emitted power should be designed tobe maximal while maintaining compliance with Class 1 laser operation.The laser light source 60 is preferably modulated at a modulationfrequency on the order of about 10 megahertz, although this frequencymay be adjusted to account for operating speeds of other subsystems inthe SRI system, and the preferred phase offsets, as mentioned earlier,are phase shifts θ in each phase image given by θ_(k)=2πκ/3; κ=0,1,2.The preferred wavelength of the laser light is about 830 nm, as thiswavelength provides an optimal balance between concerns for eye-safetyand for the typical response of the overall system as described.Although the laser light need not necessarily be uniformly distributed,a diffusion lens 66 is positioned in front of the laser light source 60in order to spread the modulated light across the desired field of viewas uniformly as possible.

An alternative light source 60 can be a plurality of amplitude-modulatedinfrared light-emitting diodes (LEDs) which are driven to generate amodulated signal according to the phase and frequency of a drive signal.The illuminator 42 also includes a standard wide-band illuminator 68that is not modulated. This illumination source is used for normalphotographic images, e.g., for the color texture image. This illuminatordevice 68 may be a commonly known and understood flash of a standardcamera system, e.g., a commonly available electronic flash of the typeuseful with photographic cameras. The illuminator 42 is connected viathe control line 64 to the controller 40, which directs the illuminator42 to operate in either of the following modes: a) a first mode in whichthe laser is operated to illuminate the scene with a plurality (bundle)of exposures, each with a unique phase offset applied to its modulatedfrequency; and b) a second mode in which the standard wide-bandilluminator 68 is turned on and the flash is initiated by the controllerduring capture of the color texture image. If ambient light issufficient, of course, it may be unnecessary for the illuminator 42 tooperate in the second mode in order to capture a color image-in thatcase, the image capture device would be instructed to operate withoutflash. Moreover, the sequence of image capture may be reversed, that is,the second mode may be engaged before the first mode or, indeed, thesecond mode might in specific situations be engaged between the severalexposures of the first mode. The illuminator 42 also communicates withthe controller 40 via the line 64 to indicate that all systems are readyfor use. While not shown, it is preferable for the system to have somevisible indicator to the operator that the laser source is powered, asthe laser is invisible to the human eye.

Referring to the modulation system 70 shown in FIG. 4, theelectromechanical grating 46 is used to modulate the reflected light 72returning from objects illuminated by the illuminator subsystem 42. Inthe preferred embodiment, the electromechanical grating 46 is aconformal grating device of the type illustrated in the aforementionedU.S. patent application Ser. No. 09/491,354, which is incorporatedherein by reference. As conceptually shown in FIG. 4 for illustrativepurposes, a pattern of elongated reflective ribbon elements 74 aresupported at their respective ends and at several intermediate supportlocations 75 (shown in broken line to indicate their location underneaththe ribbon elements) on a substrate structure 76. The center-to-centerseparation of the intermediate supports and the mechanical properties ofthe ribbon element define the mechanical resonant frequency of theconformal grating devices in their actuated state. A signal generator 82applies a corresponding modulating voltage between the substratestructure 76 and the ribbon elements 74, as a result, an electrostaticforce generated by the voltage causes the ribbon elements 74 to deformin synchronism with the modulating frequency. In this actuated state,the incident beam 78 a is diffracted into a number of discretediffracted orders. In the unactuated state, with no applied voltagedifference, the ribbon element is suspended flat between its supportsand the incident beam 78 a is primarily reflected 78 b into thedirection of the mirror. The aperture 79 allows the reflected light topass in the return beam 78 b and blocks the diffracted orders. To obtaina greater depth of modulation at the expense of the complexity of theoptical system, the reflected light can be blocked and the diffractedorders can be passed through the system. In actual practice, the area ofthe electromechanical grating 46 would be designed to fill the imagespace with thousands of ribbon elements. Furthermore, theelectromechanical grating 46 responds to light at preferred wavelengths,and is tuned to the same wavelength as the infrared light produced bythe laser source 60. Further details of the electromechanical grating,including more detailed renderings of the actual structure of thedevice, can be found in the aforementioned U.S. patent application Ser.No. 09/491,354, which is incorporated herein by reference.

The system 70 is controlled through a control line 80 attached to thesystem controller 40 and through the signal generator 82 driving theelectromechanical grating 46. (While shown as a separate component, thesignal generator 82 may be integrated together with theelectromechanical grating in a common CMOS-like element.) The signalgenerated by the signal generator 82 is in synchronization with theoverall system frequency provided by the controller 40 and serves as areference waveform that beats against the reflected waveform. Typically,the electromechanical grating 46 has a natural resonance frequencybetween 5 and 15 MHz. For an electromechanical grating 46 that issufficiently damped, the signal generator 82 periodically drives thedevice between the unactuated state and the actuated state at afrequency lower than resonance, generating light intensity modulation atthis frequency. Alternatively, for an underdamped electromechanicalgrating 46, the signal generator 82 drives the device at resonance. Inthis resonant mode of operation, the ribbon elements 74 oscillatesymmetrically about their unactuated position and generate sinusoidallight intensity modulation at twice the resonance frequency.

Since the electromechanical grating 46 is activated when a phase imageis to be captured and is inactive when a color texture image iscaptured, the retractable mirror 47 is operated to position the mirrorin-line such that one of its facets deflects the light 78 a toward thesurface of the electromechanical grating 46 when phase images are beingcaptured. The returned light 78 b is then modulated by theelectromechanical grating 46 and reflected back to the other facet ofthe retractable mirror 47, which redirects the light toward the imagecapture subsection 48. When the color texture image is being collected,the mirror is retracted sufficiently far in order to eliminate it frominterfering with the incoming light. As a result, the system is able torecord visible light necessary for the color texture image. While manydifferent types of devices may be used to toggle the mirror back andforth, FIG. 4 shows a solenoid 84 connected to the mirror 47 to drivethe mirror in the direction of the arrows 86 when so instructed by thecontroller 40. The electromechanical grating may operate either in thecontinuous modulation mode as described herein or in a pulse mode. Ifthe pulse mode operation is desired, then the image capture and rangeestimation approach follows a waveform analysis based on gating themodulation at preselected points. This has been disclosed in priordisclosures, such as U.S. Pat. No. 5,081,530 (which is incorporatedherein by reference), for range cameras and details of the approach arenot included here.

In referring to FIG. 5, the image capture subsystem 48 of the SRI deviceshares the properties common to most camera systems: a lens 90 andshutter 92 integral with the image capture subsystem (or, alternatively,the separate lens/shutter 44 shown in FIG. 1), an image capture plane 94and an image storage capability 96. As mentioned above, the imagecapture plane 94 includes a photosensor, e.g., a photosensitive film oran electronic sensor, such as a charge-coupled device (CCD). If a filmis located in the image capture plane 94, then the film itself is theimage storage capability 96. On the other hand, if an electronic sensoris located in the image capture plane 94, then the image storagecapability 96 is an electronic storage device, such as resident solidstate memory (e.g., RAM or ROM) or a removable memory (e.g., a memorycard). In addition, this subsystem 48 may also responsible forcollecting and storing some or all of the data portion of the imagebundle described above. The controller 40 of the device triggers theimage capture subsystem 48 on a control line 98. In turn the systemopens the shutter 92, records the image on the image capture plane 94and stores an image in the image storage 96 prior to a subsequentexposure period. Furthermore, the system could provide a range processorof the type shown in FIG. 6 to extract the images from the image bundlefor external processing, though this is not explicitly indicated in FIG.5. This range processor could be integral with the SRI camera or locatedin an ancillary processor.

The invention has been described with reference to a preferredembodiment. However, it will be appreciated that variations andmodifications can be effected by a person of ordinary skill in the artwithout departing from the scope of the invention.

PARTS LIST

10 range imaging system

12 scene

14 illuminator

16 modulator

18 output beam

20 reflected beam

22 receiving section

24 photocathode

26 image intensifier

30 microchannel plate

32 phosphor screen

34 capture mechanism

36 range processor

40 controller

42 illuminator

43 a output beam

43 b reflected beam

43 c optical path

46 electromechanical grating

47 mirror

48 image capture subsystem

50 image bundle

52 phase images

54 texture (color) image

56 bundle data

60 laser light source

62 modulation circuit

64 line to the controller

66 diffusion lens

68 wide-band illumination

70 modulation system

72 reflected light

74 ribbon elements

75 support locations

76 substrate structure

78 a incident beam

78 b return beam

79 aperture

80 line from the controller

82 signal generator

84 solenoid

86 direction arrow

90 lens

92 shutter

94 image capture plane

96 image storage capability

98 control line

What is claimed is:
 1. A scannerless range imaging system for capturingrange information of a scene, said system comprising; an illuminationsystem for controllably illuminating the scene with modulatedillumination of a predetermined modulation frequency, whereby modulatedillumination reflected from an object in the scene incorporates a phasedelay corresponding to a distance of the object from the range imagingsystem; an electromechanical light modulator positioned in an opticalpath of the modulated illumination reflected from the object, whereinthe electromechanical light modulator operates at a reference frequencythat corresponds to the predetermined modulation frequency andaccordingly modulates the modulated illumination reflected from theobject, thereby generating a phase image from the interference betweenthe reference frequency and the modulated illumination reflected fromthe object; and an image capture section positioned in the optical pathof the modulated illumination reflected from the object for capturingthe phase image, whereby the range information is derived from the phaseimage.
 2. The range imaging system as claimed in claim 1 wherein theelectromechanical light modulator includes modulating elements havingreflective surfaces, said system further including an optical systeminterposed in the optical path and comprised of a mirror element thatdeflects the reflected illumination upon the reflective surfaces of theelectromechanical light modulator and redirects the phase imagegenerated by the reflective surfaces to the image capture section. 3.The range imaging system as claimed in claim 2 wherein the opticalsystem is movable out of the optical path such that reflectedillumination from the object will pass directly to the image capturesection without contacting the electromechanical light modulator.
 4. Therange imaging system as claimed in claim 3 wherein the image capturesection captures at least one phase image used to derive a range imageand another image of reflected unmodulated illumination corresponding tocolor in the scene when the optical system is moved out of the opticalpath.
 5. The range imaging system as claimed in claim 1 wherein theelectromechanical light modulator is an electromechanical grating. 6.The range imaging system as claimed in claim 1 wherein the image capturesection includes a photosensitive film for capturing the phase image. 7.The range imaging system as claimed in claim 1 wherein the image capturesection includes an electronic image sensor for capturing the phaseimage.
 8. The range imaging system as claimed in claim 1 wherein theimage capture section captures a plurality of phase images correspondingto the reflected modulated illumination, wherein each phase imageincorporates the effect of the predetermined modulation frequencytogether with a phase offset unique for each image.
 9. The range imagingsystem as claimed in claim 8 wherein each unique phase offset θ is givenby θ_(i)=2πi/3; i=0,1,2.
 10. The range imaging system as claimed inclaim 8 wherein the image capture section further comprises means forstoring the phase images as a bundle of associated images.
 11. The rangeimaging system as claimed in claim 1 wherein the illumination systemincludes either a laser illuminator or an array of light emitting diodesfor producing the modulated illumination.
 12. The range imaging systemas claimed in claim 1 wherein the predetermined modulation frequency isin the infra-red spectrum.
 13. A color scannerless range imaging systemfor capturing both color and range information of a scene, said systemcomprising; an illumination system for controllably illuminating objectsin the scene with modulated illumination of a predetermined modulationfrequency, whereby modulated illumination reflected from an object inthe scene incorporates a phase delay corresponding to a distance of theobject from the range imaging system; a modulation section including (a)an electromechanical light modulator positioned in an optical path ofthe modulated illumination reflected from the object, wherein theelectromechanical light modulator operates at a reference frequency thatcorresponds to the predetermined modulation frequency and accordinglymodulates the modulated illumination reflected from the object, therebygenerating a phase image from the interference between the referencefrequency and the modulated illumination reflected from the object,wherein said phase images are used to derive the range information, and(b) an optical system located in the optical path to transmit thereflected modulated illumination to the image capture section via theelectromechanical grating when a phase image is to be captured; an imagecapture section positioned in the optical path of the modulatedillumination reflected from the object for capturing a plurality ofimages thereof, including (a) at least one phase image corresponding tothe reflected modulated illumination and (b) at least one other image ofreflected unmodulated illumination corresponding to color in the scene;and a controller for activating the modulation section when a phaseimage is to be captured and inactivating the modulation section when acolor image is to be captured.
 14. The range imaging system as claimedin claim 13 wherein the electromechanical light modulator includesmodulating elements having reflective surfaces, said optical systemfurther includes a mirror element having facets that deflect thereflected modulated illumination upon the reflective surfaces of theelectromechanical light modulator and redirect the phase image generatedby the reflective surfaces of the electromechanical light modulator tothe image capture section.
 15. The range imaging system as claimed inclaim 14 wherein the controller inactivates the modulation section bymoving the optical system out of the optical path such that thereflected unmodulated illumination corresponding to color in the scenewill pass directly to the image capture section without contacting theelectromechanical light modulator.
 16. The range imaging system asclaimed in claim 13 wherein the electromechanical light modulator is anelectromechanical grating.
 17. The range imaging system as claimed inclaim 13 wherein the image capture section includes a photosensitivefilm for capturing the phase image.
 18. The range imaging system asclaimed in claim 13 wherein the image capture section includes anelectronic image sensor for capturing the phase image and the colorimage.
 19. The range imaging system as claimed in claim 13 wherein theimage capture section captures a plurality of phase images correspondingto the reflected modulated illumination, wherein each phase imageincorporates the effect of the predetermined modulation frequencytogether with a phase offset unique for each image.
 20. The rangeimaging system as claimed in claim 19 wherein each unique phase offset θis given by θ_(i)=2πi/3; i=0,1,2.
 21. The range imaging system asclaimed in claim 19 wherein the image capture section further comprisesmeans for storing the phase images and the color image as a bundle ofassociated images.
 22. The range imaging system as claimed in claim 13wherein the illumination system includes either a laser illuminator oran array of light emitting diodes for producing the modulatedillumination.
 23. The range imaging system as claimed in claim 13wherein the predetermined modulation frequency is in the infra-redspectrum.
 24. The range imaging system as claimed in claim 13 whereinthe illumination system also emits unmodulated illumination and thereflected illumination includes unmodulated illumination originatingwith the illumination system and reflected from objects in the scene.25. The range imaging system as claimed in claim 13 wherein thereflected illumination includes unmodulated illumination from ambientillumination reflected from objects in the scene.